Papers with KV caching
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference (2026.findings-eacl)
Copied to clipboard
Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Anthony Kougkas, Xian Sun, Murali Emani, Venkatram Vishwanath, Bogdan Nicolae
| Challenge: | Large Language Models (LLMs) are exploding to large sizes, including GPT, LLaMA, and DeepSeek. |
| Approach: | They propose a fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM’s PagedAttention. |
| Outcome: | The proposed method integrates seamlessly with PagedAttention without any modifications to its CUDA attention kernels. |
Enabling Autoregressive Models to Fill In Masked Tokens (2026.findings-eacl)
Copied to clipboard
| Challenge: | Autoregressive (AR) and masked language modeling (MLM) models are incapable of mucked infilling, which is the ability to predict mangled tokens between past and future context. |
| Approach: | They propose a method that leverages the strengths of autoregressive and masked language modeling to achieve state-of-the-art mucked infilling performance. |
| Outcome: | The proposed approach outperforms existing methods on masked infilling tasks. |
XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference (2024.findings-emnlp)
Copied to clipboard
Joao Monteiro, Étienne Marcotte, Pierre-Andre Noel, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian
| Challenge: | XC-Llama uses pre-trained decoder-only models to condition generation on reference text without the prompt. |
| Approach: | They propose a model that uses cross-attention to condition generation on reference text without the prompt. |
| Outcome: | The proposed models outperform prompt-based inference methods and reduce space footprint relative to standard KV caching by two orders of magnitude. |
FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to improve latency via skipping layers have limitations . fiRST is a model-agnostic framework that reduces inference latency while maintaining quality . |
| Approach: | They propose a model-agnostic framework that skips transformer layers during decoding . it is fully compatible with KV caching, enabling faster decoding while maintaining quality . |
| Outcome: | a new framework reduces inference latency by using layer-specific routers to skip transformer layers during decoding. |